Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Fix UTF-8 handling #76

Open
wants to merge 4 commits into
base: oi
Choose a base branch
from
Open

[WIP] Fix UTF-8 handling #76

wants to merge 4 commits into from

Conversation

pyhalov
Copy link

@pyhalov pyhalov commented Mar 7, 2020

No description provided.

Copy link

@AndWac AndWac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good so far. What makes you sure that you didn't miss an open() call? There are lots of other places where open() is being called...

@pyhalov
Copy link
Author

pyhalov commented Mar 8, 2020

# Ran 1132 tests in 18490.569s - skipped 1 tests.
Skipped tests:
cli.t_pkg_help.py TestPkgHelp.test_help_character_encoding: Test Skipped: The test system must have the ja_JP.eucJP locale installed to run this test.


======================================================================
BASELINE MATCH
======================================================================

@pyhalov
Copy link
Author

pyhalov commented Mar 8, 2020

@citrus-it What do you think about utf-8 test? The only issue I see is that now you should run test suite with UTF-8 locale

@pyhalov
Copy link
Author

pyhalov commented Mar 10, 2020

@citrus-it It's not enough.
Got this exception with patched pkg:

$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/[email protected],5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) \
Traceback (most recent call last):
  File "/usr/bin/pkg", line 5968, in handle_errors
    __ret = func(*args, **kwargs)
  File "/usr/bin/pkg", line 5954, in main_func
    pargs=pargs, **opts)
  File "/usr/bin/pkg", line 2276, in install
    display_plan_cb=display_plan_cb, logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
    logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
    logger=logger, **kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
    display_plan_cb=display_plan_cb)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
    for pd in api_plan_func(**kwargs):
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
    log_op_end_all=True)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
    self._img.make_install_plan(**kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
    reject_list=reject_list)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
    self.__call_imageplan_evaluate(ip)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
    ip.evaluate()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3879, in evaluate
    self.__evaluate_pkg_plans()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4029, in __evaluate_pkg_plans
    ccancel=self.__check_cancel, progtrack=self.__progtrack)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 536, in wrapper
    return f(instance, *fargs, **f_kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1588, in prefetch_manifests
    mfstlist, excludes)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1681, in _prefetch_manifests_list
    dl_path)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1780, in _verify_manifest
    mcontent = mf.read()
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 36861: ordinal not in range(128)

@pyhalov pyhalov changed the title Fix UTF-8 handling [WIP] Fix UTF-8 handling Mar 10, 2020
@pyhalov
Copy link
Author

pyhalov commented Mar 10, 2020

It's still not complete, now it dies without exception...

$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/[email protected],5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) |
pkg: 'ascii' codec can't encode character '\u0151' in position 36861: ordinal not in range(128)

@pyhalov
Copy link
Author

pyhalov commented Mar 10, 2020

The same issue - dies on writing to /var/pkg/cache/publisher/userland/pkg/crypto%2Fca-certificates/1.0%2C5.11-2019.0.0.7%3A20200310T174048Z/manifest.file.uosjwiqq

@pyhalov
Copy link
Author

pyhalov commented Mar 10, 2020

Now more issues:

Creating Plan (Checking for conflicting actions): -
Traceback (most recent call last):
  File "/usr/bin/pkg", line 5968, in handle_errors
    __ret = func(*args, **kwargs)
  File "/usr/bin/pkg", line 5954, in main_func
    pargs=pargs, **opts)
  File "/usr/bin/pkg", line 2276, in install
    display_plan_cb=display_plan_cb, logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
    logger=logger)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
    logger=logger, **kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
    display_plan_cb=display_plan_cb)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
    for pd in api_plan_func(**kwargs):
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
    log_op_end_all=True)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
    self._img.make_install_plan(**kwargs)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
    reject_list=reject_list)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
    self.__call_imageplan_evaluate(ip)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
    ip.evaluate()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3880, in evaluate
    self.__merge_actions()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4747, in __merge_actions
    self.__find_all_conflicts()
  File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3371, in __find_all_conflicts
    offset_dict = self.image._load_actdict(self.__progtrack)
  File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 3881, in _load_actdict
    for line in of:
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5250: ordinal not in range(128)

@pyhalov
Copy link
Author

pyhalov commented Mar 10, 2020

And now we have to fix open() all over src/modules/client/image.py...

@pyhalov
Copy link
Author

pyhalov commented Mar 11, 2020

Now we can get another error:

Removing old actions                             1/1
Action install failed for 'etc/certs/CA/NetLock_Arany_(Class_Gold)_F\u0151tan\xfas\xedtv\xe1ny.pem' (pkg://userland/crypto/ca-certificates):
  UnicodeEncodeError: 'ascii' codec can't encode character '\u0151' in position 42: ordinal not in range(128)

@Toasterson
Copy link

@pyhalov With #93 could we merge this or are we still stuck on one case where it still calls open without encoding?

@pyhalov
Copy link
Author

pyhalov commented Nov 3, 2021

Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?

@Toasterson
Copy link

Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?

No we needed to remove C locale and switch to C.UTF-8 one should not call IPS with just the C locale. We could make a patch to the python source to fix the if case that checks if the string contains .UTF-8 in locale and simply tell it to assume UTF-8 aswell in those cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants